Automatic reconfiguration of high performance computing job schedulers based on user behavior, user feedback, and job performance monitoring

ABSTRACT

A job in a high performance computing system is executed according to a current scheduler configuration. The job is monitored to determine the job&#39;s actual resource usage, the job&#39;s performance metric and the job&#39;s completion. Responsive to determining that the job completed executing, a difference is computed between allocated resources allocated according to the current scheduler configuration at job submission for execution on the high performance computing system and the job&#39;s actual resource usage at the job&#39;s completion, wherein the difference is computed for each of N number of jobs that complete executing on the high performance computing system, wherein all of the N number of jobs are associated with a given user. Responsive to determining that the difference for said each of N number of jobs meets a criterion, the current scheduler configuration is automatically reconfigured to reduce the allocated resources for the jobs associated with the given user.

FIELD

The present application relates generally to computers, and moreparticularly to high performance computing systems, and resourcescheduling on high performance computing systems.

BACKGROUND

High performance computing (HPC) clusters are managed by job schedulersthat are configured with policies that define how users access resourcesto run their applications. Job scheduling policies and theirconfiguration aim to achieve goals such as throughput, response time,utilization levels, and meeting deadlines. Generally, job schedulersrequire a human expertise to be configured for achieving goodperformance. Policies may influence user behavior with respect to jobsubmission and required resources on HPC clusters. For example, HPCclusters have queues for jobs that take short duration to execute, forexample jobs that run for less than 6 hours. These queues usually havehigher priority than queues designated to hold jobs that take longerduration, for example, jobs that run for more than 10 hours. Given suchpolicies with respect to different types of queues, users may prefer tosubmit their jobs to the queue with higher priority, for example, thoseholding short duration jobs. Users may also try to divide a job thatgenerally takes longer duration to execute into smaller pieces, so thatthose jobs can be submitted to the short job queue, and therefore,receive priority in running those jobs. Another example policy mayinclude allowing a job queue to have access to specialized resourcessuch Graphical Processing Units. Based on such policy, a user may preferone queue over another queue when submitting a job. Despite thepolicies, however, resources may be wasted, which in turn increases useof energy, requires additional acquisition and maintenance of machines,and delays user access to the machines.

BRIEF SUMMARY

A method and system of automatically reconfiguring a high performancecomputing job scheduler may be provided. The method, in one aspect, mayinclude executing a job in a high performance computing system accordingto a current scheduler configuration, the job associated with a givenuser. The method may also include monitoring the executing job todetermine the job's actual resource usage, the job's performance metricsand the job's completion. The method may further include monitoring aproblem ticket system for an incoming problem ticket associated with thejob. The method may also include, responsive to detecting the incomingproblem ticket, reconfiguring the current scheduler configuration toincrease one or more resources in the high performance computing systemfor submitted jobs associated with the given user. The method may alsoinclude, responsive to determining that the job completed executing,computing a difference between allocated resources allocated accordingto the current scheduler configuration at job submission for executionon the high performance computing system and the job's actual resourceusage at the job's completion, wherein the difference is computed foreach of N number of jobs that complete executing on the high performancecomputing system, wherein all of the N number of jobs are associatedwith a given user. The method may also include, responsive todetermining that the difference for each of N number of jobs meets acriterion, reconfiguring the current scheduler configuration to reducethe allocated resources for the submitted jobs associated with the givenuser.

A high performance computing job scheduler system, in one aspect, mayinclude at least one hardware processor executing a job in a highperformance computing system according to a current schedulerconfiguration, the job associated with a given user. A storage devicemay be coupled to the at least one hardware processor and storescheduler configurations, from which the at least one hardware processorretrieves the current scheduler configuration. The at least one hardwareprocessor may monitor the executing job to determine the job's actualresource usage, the job's performance metric and the job's completion.The at least one hardware processor may monitor a problem ticket systemfor an incoming problem ticket associated with the job. Responsive todetecting the incoming problem ticket, the at least one hardwareprocessor may reconfigure the current scheduler configuration toincrease one or more resources in the high performance computing systemallocated for submitted jobs associated with the given user. Responsiveto determining that the job completed executing, the at least onehardware processor may compute a difference between allocated resourcesallocated according to the current scheduler configuration at jobsubmission for execution on the high performance computing system andthe job's actual resource usage at the job's completion, wherein thedifference is computed for each of N number of jobs that completeexecuting on the high performance computing system. Responsive todetermining that the difference for said each of N number of jobs meetsa criterion, the at least one hardware processor may reconfigure thecurrent scheduler configuration to reduce the allocated resources, e.g.,for the submitted jobs associated with the given user, and update thescheduler configurations stored in the storage device according to thereconfiguring.

A computer readable storage medium storing a program of instructionsexecutable by a machine to perform one or more methods described hereinalso may be provided.

Further features as well as the structure and operation of variousembodiments are described in detail below with reference to theaccompanying drawings. In the drawings, like reference numbers indicateidentical or functionally similar elements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram illustrating a method of the present disclosurein one embodiment.

FIG. 2 is a diagram showing components of a system of the presentdisclosure in one embodiment.

FIG. 3 illustrates a schematic of an example computer or processingsystem that may implement a job scheduling system in one embodiment ofthe present disclosure.

DETAILED DESCRIPTION

A system and method may be provided that configure High PerformanceComputing (HPC) job (or process) queues based on user behavior, userfeedback and actual usage of resources determined by job performancemonitoring. The system and method may optimize one or more jobschedulers of High Performance Computing (HPC) resources, for example,may implement a job scheduler component (e.g., a software component)that contains configurations and/or policies on how users (e.g., userapplications) share computing resources to achieve a goal (e.g.,response time or resource utilization) and receives user requests toaccess such resources. A job includes specification on how to run anapplication and specification of resources (e.g., network, processor,memory, and storage) that impact job performance. A job specification,for example, may indicate a number of processors, memory size, diskspace size and type of communication network standard, for example: 12processors, 4 GB of memory, 100 GB of disk space, and InfiniB andNetwork connecting all processors. As an example, consider Platform LoadSharing Facility (LSF), a scheduler commercialized by InternationalBusiness Machines Corporation (IBM), Armonk, N.Y. In LSF a systemadministrator (sysadmin) may configure queues for different purposes.For example, an HPC cluster may be configured to have queues that havejobs that are only scheduled at night. In this example, although a usercan submit jobs to this queue at any moment, the cluster will onlyexecute them at night. As another example, suppose a user wants tosubmit a job that runs on all hosts of a cluster. The user can submitsuch a job by specifying that the user wants to use all processorsavailable. A policy in the system may disallow this request on businessdays, but allow the request on weekends. The scheduler according to thepolicy will wait until a weekend to effectively execute such a job. Ifthe cluster does not complete the job during the weekend, the schedulersuspends the user job until the next weekend.

The following shows an example queue configuration for a default queue(for Spectrum LSF):

Begin Queue QUEUE_NAME = normal PRIORITY = 30 NICE = 0 INTERACTIVE = NOFAIRSHARE = USER_SHARES[[default,1]] End Queue

The above queue configuration specifies a non-interactive queue (onlyruns batch jobs) with “low” priority. The scheduling priority isfairshare and the scheduler favors users that run few jobs over usersthat run many jobs.

The example below illustrates a queue configuration specified forinteractive jobs. Since users have to be on the terminal to interactwith their jobs, priority is higher. Also, this example queue onlysupports interactive jobs while still using fairshare scheduling:

Begin Queue QUEUE_NAME = interactive PRIORITY = 40 NICE = 0 INTERACTIVE= ONLY NEW_JOB_SCHED_DELAY = 0 FAIRSHARE = USER_SHARES[[default,1]] EndQueue

A user of a system that includes the above example queue configurationsmay submit a normal job by entering the following command on a terminal:

$ bsub job

If a job is interactive, the user can submit the job with the “−I”switch, such as:

$ bsub −q interactive −I job

If the job is not interactive but the user tries to submit it to theinteractive queue, the scheduler policy denies the job since theinteractive queue only accepts interactive jobs and the scheduler doesnot submit the job.

The job scheduler component in one embodiment monitors the differencebetween user requested and actual resource usage in a job that isexecuted. Such difference is defined as user behavior. The job schedulercomponent in one embodiment also tracks the job's performance on aresource set. For instance, batch schedulers have capability todetermine such differences. Systems commands may be invoked to determineactual resource usage. As an example, executing the command “time”,available on UNIX-like reports the time a program consumed computing(user time), how much time it performed in the kernel performingInput/Output (I/O) operations, for example (system time), and the totaltime the program spent overall. Consider, the example below:

$ time ./a.out

./a.out 0,14s user 0,00s system 98% cpu 0,145 total

According to the output, the command above (a.out) took 145 ms to run.Of these, 140 ms was in doing computation and the other 5 ms doingmemory allocations. In the above example command line, the “time”command calls the program (a.out) and then calls “wait4”, which is ablocking system call that only returns when the child process finishes.When the child is finished, a data structure called “struct rusage” isfilled with usage data that, among other things, includes the timesdescribed above. When the scheduler starts a job, the scheduler may callsuch command, e.g., “time”, and once the job is completed, the schedulercan compare the requested run time with the actual run time, e.g., asubtraction (requested time−actual time). Other system commands may beinvoked to determine other resource usage and compute the differencebetween the requested and actual resource usage.

The job scheduler monitoring component in one embodiment automaticallymodifies a scheduler configuration and/or policy to adapt the behaviorof the job scheduler. The scheduler configuration and/or policy may becustomized per user or per a plurality of users (e.g., a user group),e.g., based on (i) the behavior of jobs based on requested versus actualresource needs, and the (ii) user perception of impact on jobs'performance and response time due to scheduler reconfiguration.

HPC job management systems, such as IBM Spectrum LSF, have the notion ofuser groups and user specific profiles for setting up personalizedlimits and policies. Once groups have been defined, systemsadministrators can configure specialized queues with different behaviorsfor different users. The same is valid for hosts and host groups. Asystem can be configured in a way such that, for example, only specificusers can submit jobs to specific hosts. The following shows examplesfor a default LSF configuration:

Begin UserGroup

GROUP_NAME GROUP_MEMBER USER_SHARES #GROUP_ADMIN #ugroup1 (user1 user2user3 user4) ([user1, 4] [others, 10]) #(user1 user2[full]) #ugroup2(all) ([user3, 10] [others, 15]) #(user3[usershares]) #ugroup3 (ugroup1user5 user6 ([user5, 16] [user6, 34] #( ) user7) [user7, 15] [ugroup1,40]) End UserGroup

Begin User

USER_NAME MAX_PEND_JOBS #user1 800 # user1 has pend threshold of 800#ugroup1 @ 500 # each user in ugroup1 has threshold of 100 #ugroup1 1000# collectively ugroup1 has threshold of 1000 #default 100 # default, anyuser/group has threshold of 100 End User

User feedback may be also considered in customizing the schedulerconfiguration and/or policy. User perception of impact can be added tothe system by the User Interface component 206, in which users canprovide their perception of their job's response time. Response time isdefined as the time it took between job submission and completion and,includes the amount of time a job spent in the scheduler queue plus thetime it took for the job to actually execute. The scheduler may measuresuch times by querying the system clock upon job submission, start ofexecution and end of execution.

FIG. 1 is a flow diagram illustrating a method of the present disclosurein one embodiment. At 102, jobs are executed according to a previousscheduler configuration of queues and policies. The flow diagram shownin FIG. 1 describes a process that is executed periodically, with thefrequency, for example, defined by the systems administrators. Since theprocess starts with a previous configuration, the initial schedulerconfiguration may come from the default configuration that an HPCscheduler uses, or from the systems administrator customization of thesystem prior to the execution of the system in the present embodiment.

At 104, the job scheduler component monitors its job submission rate,computed from the number of jobs submitted per unit of time, jobcompletion rate, computed from the number of jobs finished per unit oftime, and job performance, the time it took for a job to finish. When ajob completes, the system can gather information about the job for eachmachine it executed on. Information includes (e.g., populated in thestruct rusage structure in UNIX-based operating systems) user time,system time, maximum resident size memory usage, amount of sharing oftext segment memory with other processes, amount of data segment memoryused, amount of stack memory used, number of soft page faults (thoseserviced by reclaiming a page from the list of pages awaitingreallocation), hard page faults (those that required I/O), number oftimes a process was swapped out of physical memory, number of inputoperations via the file system, number of output operations via the filesystem, number of Inter-Process Communication messages sent, number ofInter-Process Communication messages received, number of signalsdelivered, number of voluntary context switches (usually to wait forresources to be available), and number of involuntary context switches(usually because higher priority processes became runnable or thecurrent process used up its time slice).

Ticketing system and user status commands are also monitored. Forexample, incoming tickets in a problem ticket system may be monitored todetermine whether any of the problem tickets pertain to performancedegradation in the jobs that are scheduled and executed according to thescheduler configuration specified at 102. For example, the monitoringtakes place by having a daemon—a server process—periodically consultingthe company's ticketing system and searching for performance degradationtickets. Such identification may be done, for example, by searching fora specific “degradation” category in the ticket system or by using textextraction algorithms. Once new entries are detected in the ticketsystem database, the monitoring process can check whether any of theticket's author jobs have been affected by changes in the schedulingsystem—in a previous configuration, the job would have been scheduled ina determined way, whereas with the new configuration the job may havebeen scheduled in a different way.

At 106, if it is determined that a problem ticket is created because ofa system performance degradation caused by a scheduling systemreconfiguration, the scheduler configuration may be restored to aprevious state. In case the system is using the default configuration(initial configuration at the first iteration before any configurationchanges), there is no state to return to, as, by definition, performancehas not degraded. The restoration to a previous state can be made bymaintaining a list of all the changes made to the schedulerconfiguration. For example, the changes made to the schedulerconfiguration may be maintained in a source control system. Thescheduler configuration that is changed or restored is for a givenuser's jobs (e.g., those submitted by the given user), wherein theproblem ticket that is detected is related to a job of the given user.For example, from the detected problem ticket, a user associated withthe job that is the subject of the problem ticket may be determined, andthe scheduler configuration may be changed for jobs that are submittedby that user.

In one embodiment, the flow at 104 is executed asynchronously andperiodically. From it, two parallel flows can start, one going to 106,and another going to 108-124. In one embodiment, a monitoring daemonperiodically checks the system performance and support tickets at 104,and at 106 the monitoring daemon checks whether performance degradationsand user support tickets were created.

At 108, for each job that is completed (determined from monitoring thejob completion at 104), the processing at 110, 112, 114, 116, 118,120,122, and 124 may be performed.

At 116, user behavior over time, e.g., the amount of resources requestedfor each job (the user submits) and the actual resource usage, isclustered by a clustering algorithm, such as k-means or Gaussian MixtureModels. These clustering algorithms can use resource usage levelsconsidering central processing unit (CPU), memory, network, and diskspace.

At 118, based on created clusters, the job scheduler reconfigures itsconfiguration, for example, adds, removes, and/or modifies a job queuethat includes scheduled jobs.

In one embodiment, the processing flow from 108 to 116 may be performedin parallel with the process flow from 108 to 110, for example, 116 and110 may be executed in any order, or in parallel.

At 110, for the job that is being considered (a completed job at 108),the actual resource usage by the job and the allocated (requested)resource for the job are compared. Run-time performance of the job isalso compared by measuring the new execution time and comparing it witha previously-recorded execution time (if it exists).

At 112, it is determined whether the allocated (requested) resources andthe actual resource usage are substantially similar. For instance, ifthe difference between the actual resource usage by the job and theallocated (requested) resource for the job is within a threshold value,it may be determined that the allocated (requested) resources and theactual resource usage are substantially similar. For example, requestedresources−used resources=0 or another defined minimal value. In thiscase, good behavior (not-overestimated behavior) is found, otherwise,the behavior may be defined as overestimated behavior. In oneembodiment, a predefined number of jobs (N jobs) may be considered indetermining whether user behavior should be defined to be overestimated.For example, only if after N number or more of jobs are all found tohave the difference values that are greater than the threshold value(e.g., 0 or another defined minimal value), the user behavior may bedefined to be overestimated.

If it is determined that the user behavior is overestimated (e.g., if itis determined that the allocated (requested, e.g., as a result of userrequesting) resources are larger than the actual used resources (e.g.,requested resources−used resources>0 or the threshold value) for one ora defined N number of jobs), then at 114, the scheduler configuration isreconfigured to reduce the resources for this user, for example, forjobs submitted by this user. For instance, the resources may be reducedto the actual used resources, or example a percentage or fraction of thedifference.

At 120, based on the comparison at 110, if it is determined that asystem reconfiguration in a previous iteration for this user affectedperformance, a previous configuration may be reverted. If the previousconfiguration did not affect the job performance negatively, no actionis taken, and the logic may return, for example to 108 to processanother job performance.

At 122, the previous configuration is reverted. For example, a systemrecorded history of changes may be accessed, and the configuration mayrevert back to a configuration immediately before the currentconfiguration.

The resources may be reduced at a ratio, for example, at a predefined Xratio based on application performance of a previous execution. Thatway, in one embodiment, if a user was able to allocate 100 processors ina previous execution, but that was an over-estimation (over-estimateduser behavior) and the system is configured to decrease access by 10%,then the maximum number of processors this user is able to allocate maybe reduced to 90. Additionally, other resources can be verified, such asrequested memory usage, number of Graphics Processing Units, and thenumber of jobs that can be submitted to the system at a single time.

At 124, the job scheduler component may notify the user that theresources have been reduced. In another embodiment, the job schedulermay reduce the resources without notifying the user.

The processing performed according to the method shown in FIG. 1 may beexecuted continuously as the HPC system schedules and executes jobscontinuously. For instance, the initial scheduler configuration shown at102 would represent a current configuration used in the currentiteration, for example, configured from an immediate previous iterationof the method.

FIG. 2 is a diagram showing components of a system of the presentdisclosure in one embodiment. A user 202 may perceive priority of a jobaccording to how important the user thinks the user's jobs are relativeto the user's peers and also relative to how a system administration 222may think. Job ratings allow users to rank and rate each others' jobs (atechnique known as collaborative filtering) when a system is busy. Arecommendation system may allow user to input information such as whichqueues and/or hosts deliver effective results as perceived by the user.With such ratings, the recommendation system can use the collaborativefilter to rank jobs according to the value perceived by users, allowingscheduler reconfigurations to be more relevant to users. The system mayreceive input, for example, via a ticketing system that stores in adatabase 220 problem tickets generated by users or system generatedproblem tickets, and also via a user interface 206 displayed on adisplay device 204. A user may specify via a user interface 206information such as “my job is 30% slower than usual” and “my job isaborting after 5 minutes of execution”.

A job scheduler 208 may be executing on one or more hardware processors224, and may include a user behavior and job performance monitor orfunctionality 210, a user behavior classifier and behavior changedetector or functionality 212, and a scheduler policy reconfigurator orfunctionality 214. The user behavior and job performance monitor orfunctionality 210 may monitor executing jobs and their performance. Thejobs are executed according to a scheduling configuration and policy 218stored in a database, for example, stored in a storage device. At thecompletion of each job, the user behavior and job performance monitor210 may compute or determine the user behavior associated with the job(e.g., the difference between the requested or allocated resources andthe actual resources used by the job). The performance of the job, forexample, throughput, speed and other performance related data isdetermined. If no problems have been registered via the ticket system,or the job completes successfully (e.g., detected by a clustermanagement system automatically), then the performance of the job isconsidered to be satisfactory. The user behavior (determined based onthe difference between the requested and actual usage of resources) andthe performance determined for the job is stored in a job database 216,e.g., historical job data database, stored in a storage device. As anexample, data such as CPU, memory, disk, and network utilizationassociated with executing jobs may be monitored and determined. Thisdata is the actual utilization of the resources, which is comparedagainst what the user requested. The difference determines whether auser's behavior is an overestimated behavior or not-overestimatedbehavior.

The user behavior classifier and behavior change detector 212 maydetermine or compute user behavior. For example, user behaviorclassifier and behavior change detector 212 computes the differencebetween the allocated resources requested by a user and the actualresources used by the job. If N number of jobs (all associated with theuser) has the difference value greater than zero or another definedthreshold value, the user behavior classifier and behavior changedetector 212 may classify the user behavior as being overestimated or anoverestimation. N may be configurable, and may be defined as any numberbetween 1 to a defined or configure positive integer. This componentalso detects behavior changes. For example, if the user usually asks for12 processors, but the CPU utilization level of the user's job is 50% itmeans this user's behavior is overestimated behavior (the useroverestimated the resources required to run the user's job). If theuser's job starts to request 8 processors, and the CPU utilization leveljumps to 80%, the behavior is improving, and the change in behavior (inthis example, improvement) is detected by the “change detector”.

The scheduler policy reconfigurator 214 may reconfigure the currentscheduling configuration and/or policy 218 related to jobs associatedwith the user, responsive to determining that the requested or allocatedresources for the job was an overestimation of the needed resources, forexample, the amount of the requested or allocated resources is largerthan the actual resources used by the job by more than a thresholdamount (e.g., for N number of jobs).

In one aspect, the scheduler 208 may also determine by monitoring aticketing system's tickets 220, whether the current schedulingconfiguration/policies are an underestimation. For instance, if ticketsare generated during the execution of the jobs according to the currentscheduling configuration and/or policies, the tickets indicatingproblems related to computing resources and performance of the job, thescheduler 208, for instance, via the scheduler policy reconfigurator214, reconfigure the current scheduling configuration and/or policies toincrease the resources. For example, if the user usually asks for 12processors, but the CPU utilization level of the user's job is 50%, thismay mean that the user overestimated the resources required to run thejob, implying behavior that is overestimated. If the user's job startsto request 4 processors, and the CPU utilization level jumps to 100%,the user may detect poor performance in executing the job because theresources may be underestimated. In this case, the user may, forinstance, generate in the ticket system a problem ticket specifying thatthe performance of the computing resources is poor. As another example,the system may automatically generate a ticket specifying poorperformance, responsive to detecting that the job's performance is notup to a standard.

FIG. 3 illustrates a schematic of an example computer or processingsystem that may implement a job scheduling system in one embodiment ofthe present disclosure. The computer system is only one example of asuitable processing system and is not intended to suggest any limitationas to the scope of use or functionality of embodiments of themethodology described herein. The processing system shown may beoperational with numerous other general purpose or special purposecomputing system environments or configurations. Examples of well-knowncomputing systems, environments, and/or configurations that may besuitable for use with the processing system shown in FIG. 3 may include,but are not limited to, personal computer systems, server computersystems, thin clients, thick clients, handheld or laptop devices,multiprocessor systems, microprocessor-based systems, set top boxes,programmable consumer electronics, network PCs, minicomputer systems,mainframe computer systems, and distributed cloud computing environmentsthat include any of the above systems or devices, and the like.

The computer system may be described in the general context of computersystem executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.The computer system may be practiced in distributed cloud computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed cloudcomputing environment, program modules may be located in both local andremote computer system storage media including memory storage devices.

The components of computer system may include, but are not limited to,one or more processors or processing units 12, a system memory 16, and abus 14 that couples various system components including system memory 16to processor 12. The processor 12 may include a module 30 that performsthe methods described herein. The module 30 may be programmed into theintegrated circuits of the processor 12, or loaded from memory 16,storage device 18, or network 24 or combinations thereof.

Bus 14 may represent one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus.

Computer system may include a variety of computer system readable media.Such media may be any available media that is accessible by computersystem, and it may include both volatile and non-volatile media,removable and non-removable media.

System memory 16 can include computer system readable media in the formof volatile memory, such as random access memory (RAM) and/or cachememory or others. Computer system may further include otherremovable/non-removable, volatile/non-volatile computer system storagemedia. By way of example only, storage system 18 can be provided forreading from and writing to a non-removable, non-volatile magnetic media(e.g., a “hard drive”). Although not shown, a magnetic disk drive forreading from and writing to a removable, non-volatile magnetic disk(e.g., a “floppy disk”), and an optical disk drive for reading from orwriting to a removable, non-volatile optical disk such as a CD-ROM,DVD-ROM or other optical media can be provided. In such instances, eachcan be connected to bus 14 by one or more data media interfaces.

Computer system may also communicate with one or more external devices26 such as a keyboard, a pointing device, a display 28, etc.; one ormore devices that enable a user to interact with computer system; and/orany devices (e.g., network card, modem, etc.) that enable computersystem to communicate with one or more other computing devices. Suchcommunication can occur via Input/Output (I/O) interfaces 20.

Still yet, computer system can communicate with one or more networks 24such as a local area network (LAN), a general wide area network (WAN),and/or a public network (e.g., the Internet) via network adapter 22. Asdepicted, network adapter 22 communicates with the other components ofcomputer system via bus 14. It should be understood that although notshown, other hardware and/or software components could be used inconjunction with computer system. Examples include, but are not limitedto: microcode, device drivers, redundant processing units, external diskdrive arrays, RAID systems, tape drives, and data archival storagesystems, etc.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements, if any, in the claims below areintended to include any structure, material, or act for performing thefunction in combination with other claimed elements as specificallyclaimed. The description of the present invention has been presented forpurposes of illustration and description, but is not intended to beexhaustive or limited to the invention in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the invention.The embodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

This listing of claims will replace all prior versions, and listings, ofclaims in the application:

1.-7. (canceled)
 8. A computer readable storage device storing a programof instructions executable by a machine to perform a method ofautomatically reconfiguring a high performance computing job scheduler,the method comprising: executing a job in a high performance computingsystem according to a current scheduler configuration, the jobassociated with a given user; monitoring the executing job to determinethe job's actual resource usage, the job's performance metric and thejob's completion; monitoring a problem ticket system for an incomingproblem ticket associated with the job; responsive to detecting theincoming problem ticket, reconfiguring the current schedulerconfiguration to increase one or more resources in the high performancecomputing system allocated for submitted jobs associated with the givenuser ; responsive to determining that the job completed executing,computing a difference between allocated resources allocated accordingto the current scheduler configuration at job submission for executionon the high performance computing system and the job's actual resourceusage at the job's completion, wherein the difference is computed foreach of N number of jobs that complete executing on the high performancecomputing system; and responsive to determining that the difference forsaid each of N number of jobs meets a criterion, reconfiguring thecurrent scheduler configuration to reduce the allocated resources forthe submitted jobs associated with the given user.
 9. The computerreadable storage device of claim 8, wherein the criterion includes thedifference being greater than a predefined threshold value.
 10. Thecomputer readable storage device of claim 8, further comprisingnotifying the given user associated with the job that the allocatedresources for the job have been reduced.
 11. The computer readablestorage device of claim 8, wherein N is a configurable positive integer.12. The computer readable storage device of claim 11, wherein N is one.13. The computer readable storage device of claim 11, wherein N isgreater than one.
 14. The computer readable storage device of claim 8,wherein responsive to determining that the job's performance metric isdegraded, reconfiguring the current scheduler configuration to increasethe allocated resources.
 15. A high performance computing job schedulersystem comprising: at least one hardware processor executing a job in ahigh performance computing system according to a current schedulerconfiguration, the job associated with a given user; a storage devicecoupled to the at least one hardware processor and storing schedulerconfigurations, from which the at least one hardware processor retrievesthe current scheduler configuration; the at least one hardware processormonitoring the executing job to determine the job's actual resourceusage, the job's performance metric and the job's completion; the atleast one hardware processor monitoring a problem ticket system for anincoming problem ticket associated with the job; responsive to detectingthe incoming problem ticket, the at least one hardware processorreconfiguring the current scheduler configuration to increase one ormore resources in the high performance computing system allocated forsubmitted jobs associated with the given user; responsive to determiningthat the job completed executing, the at least one hardware processorcomputing a difference between allocated resources allocated accordingto the current scheduler configuration at job submission for executionon the high performance computing system and the job's actual resourceusage at the job's completion, wherein the difference is computed foreach of N number of jobs that complete executing on the high performancecomputing system; and responsive to determining that the difference forsaid each of N number of jobs meets a criterion, the at least onehardware processor reconfiguring the current scheduler configuration toreduce the allocated resources for the submitted jobs associated withthe given user, and updating the scheduler configurations stored in thestorage device according to the reconfiguring.
 16. The system of claim15, wherein the criterion includes the difference being greater than apredefined threshold value.
 17. The system of claim 15, wherein the atleast one hardware processor notifies the given user associated with thejob that the allocated resources for the job have been reduced.
 18. Thesystem of claim 15, wherein N is a configurable positive integer. 19.The system of claim 18, wherein N is one.
 20. The system of claim 15,wherein responsive to determining that the job's performance metric isdegraded, the at least one hardware processor reconfiguring the currentscheduler configuration to increase the allocated resources.